Context-based identification of anomalous log data

ABSTRACT

Disclosed herein are methods, systems, and processes for context-based identification of anomalous log data. Log data with multiple original logs is received at an anomalous log data identification system. A context associated training dataset is generated by splitting a string in a log into multiple split strings, generating a context association between each split string and a unique key that corresponds to the log, and generating an input/output (I/O) string data batch that includes I/O string data for each split string in the log by training each split string against every other split string in the log. A context-based anomalous log data identification model is then trained according to a machine learning technique using the I/O string data batch that includes a list of unique strings in the context associated training dataset. The training tunes the context-based anomalous log data identification model to classify or cluster a vector associated with a new string in a new log that is not part of the multiple original logs as anomalous.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims benefit of priority to U.S. ProvisionalPatent Application No. 62/947,032 filed on Dec. 12, 2019 titled “LogAnomaly Detection, Analysis, and Visualization,” the disclosure of whichis hereby incorporated by reference as if set forth in its entiretyherein.

BACKGROUND Field of the Disclosure

This disclosure is related to context-based identification of anomalouslog data in managed detection and response (MDR) computing systems.

Description of the Related Art

Security analysts in a security operations center (SOC) monitorcomputing devices and computing networks for known malicious behavior.Such monitoring involves identifying existing and/or on-going maliciousactivity (called threat detection) and also investigating novel and/orundetected malicious activity (called threat hunting). Threat hunting isa crucial part of a security analyst's role in a SOC as it helpsidentify and alert to new malicious behavior and/or advanced threatsfrom hackers and other types of malicious actors.

In several current SOC implementations, threat hunting requires theretrieval and analysis of large amounts of forensic data from acustomer's computing environment. For example, voluminous forensic datahas to be first stored in a database and the database has to then bemanually manipulated in an attempt to identify outliers and anomalies.Unfortunately, threat hunting in at least this manner is not onlylaborious, time-intensive, and computing- and human-resourceprohibitive, but is also not efficient or entirely accurate. Performingthreat hunting with existing procedures like stacking, among others,presents a significant cybersecurity risk and redundant costs to modernenterprises.

SUMMARY OF THE DISCLOSURE

Disclosed herein are methods, systems, and processes for context-basedidentification of anomalous log data in managed detection and response(MDR) computing systems. One such method involves receiving log datacomprising several logs and generating a context associated trainingdataset by splitting a string in a log of the several logs into multiplesplit strings, generating a context association between each splitstring and a unique key that corresponds to the log, and generating aninput/output (I/O) string data batch that includes I/O string data foreach split string in the log by training each split string against everyother split string in the log. The method then includes training acontext-based anomalous log data identification model using the I/Ostring data batch that includes a list of unique strings in the contextassociated training dataset according to a machine learning technique.The training tunes the context-based anomalous log data identificationmodel to classify or cluster a vector associated with a new string in anew log that is not part of the plurality of logs as anomalous.

In one embodiment, the method involves generating a dense vector for thelog by accessing the list of unique split strings and averaging severalvectors that include at least one vector for each unique split string inthe list of unique split strings. The dense vector indicates a mappingof each unique split string in the list of unique split strings to thedense vector being trained. In another embodiment, the context-basedanomalous log data identification model is also trained with additionalI/O string data generated by the context-based anomalous log dataidentification system for each log of the several logs. In someembodiments, the log data includes process information associated withcomputing systems generating the log data and the process informationincludes several process names/hashes.

In certain embodiments, training the context-based anomalous log dataidentification model to perform cluster analysis is based on whether anexecutable that is part of the process information is a good executablethat is part of a bad path. The good executable and the bad path arepre-identified based at least on a classifier prior to performing thecluster analysis. In other embodiments, training the context-basedanomalous log data identification model to perform cluster analysis isbased at least on a number of occurrences of a process name/hash of theplurality of process names/hashes in the log.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations and omissions of detail; consequentlythose skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, features, and advantages of the present disclosure, as definedsolely by the claims, will become apparent in the non-limiting detaileddescription set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be better understood, and its numerousobjects, features and advantages made apparent to those skilled in theart by referencing the accompanying drawings.

FIG. 1A is a block diagram 100A of a managed detection and response(MDR) server, according to one embodiment of the present disclosure.

FIG. 1B is a block diagram 100B of a MDR server that implements a loganomaly detection engine, a user, hostname, process (UHP) engine, aMarkovian analysis engine, and a security operations engine, accordingto one embodiment of the present disclosure.

FIG. 2 is a block diagram 200 of a word embedding model for naturallanguage processing (NLP), according to one embodiment of the presentdisclosure.

FIG. 3A is a block diagram 300A of a log shown as continuous text,according to one embodiment of the present disclosure.

FIG. 3B is a block diagram 300B of a log with a window size smaller thanthe log, according to one embodiment of the present disclosure.

FIG. 4 is a block diagram 400 of a modified word embedding model,according to one embodiment of the present disclosure.

FIG. 5 is a block diagram 500 of summed word vectors, according to oneembodiment of the present disclosure.

FIG. 6 is a block diagram 600 of a log anomaly detection system,according to one embodiment of the present disclosure.

FIG. 7 is a block diagram 700 of pre-processed logs and outputs ofmodified logs, field strings, and training strings, according to oneembodiment of the present disclosure.

FIG. 8 is a block diagram 800 of a training filter, according to oneembodiment of the present disclosure.

FIG. 9 is a block diagram 900 of a word embedding implementation,according to one embodiment of the present disclosure.

FIG. 10 is a block diagram 1000 of word vectors, according to oneembodiment of the present disclosure.

FIG. 11 is a block diagram 1100 of a field vector implementation,according to one embodiment of the present disclosure.

FIG. 12 is a block diagram 1200 of field vectors, according to oneembodiment of the present disclosure.

FIG. 13 is a block diagram 1300 of a log vector implementation,according to one embodiment of the present disclosure.

FIG. 14 is a flowchart 1400 of a process for detecting anomalous logdata, according to one embodiment of the present disclosure.

FIG. 15 is a flowchart 1500 of a process for reorganizing original logdata, according to one embodiment of the present disclosure.

FIG. 16 is a flowchart 1600 of a process for generating a model with aword embedding tensor, according to one embodiment of the presentdisclosure.

FIG. 17 is a flowchart 1700 of a process for combining field vectorsinto a field embedding tensor, according to one embodiment of thepresent disclosure.

FIG. 18 is a flowchart 1800 of a process for identifying anomalies inlog data, according to one embodiment of the present disclosure.

FIG. 19A is a block diagram 1900A of a user interface of a UHP-TimelineVisualizer, according to one embodiment of the present disclosure.

FIG. 19B is a block diagram 1900B of a UI-based visualization ofanomalous log data, according to one embodiment of the presentdisclosure.

FIG. 20 is a block diagram 2000 of a process for preparing and sendingvisualization data to a UI-based web application, according to oneembodiment of the present disclosure.

FIG. 21 is a block diagram 2100 of a Markovian analysis engine,according to one embodiment of the present disclosure.

FIG. 22 is a flowchart 2200 of a process for identifying anomalous logdata using Markovian prediction models, according to one embodiment ofthe present disclosure.

FIG. 23 is a flowchart 2300 of a process to perform log anomalydetection, according to one embodiment of the present disclosure.

FIG. 24 is a block diagram 800 of a computing system, illustrating a loganomaly detection, analysis, and visualization (DAV) engine implementedin software, according to one embodiment of the present disclosure.

FIG. 25 is a block diagram 900 of a networked system, illustrating howvarious devices can communicate via a network, according to oneembodiment of the present disclosure.

While the disclosure is susceptible to various modifications andalternative forms, specific embodiments of the disclosure are providedas examples in the drawings and detailed description. It should beunderstood that the drawings and detailed description are not intendedto limit the disclosure to the particular form disclosed. Instead, theintention is to cover all modifications, equivalents and alternativesfalling within the spirit and scope of the disclosure.

DETAILED DESCRIPTION

Example of Context-Based Identification of Anomalous Log Data

As shown in FIG. 1A, managed detection and response (MDR) server 105with log anomaly detection engine 110 performs context-basedidentification of anomalous log data in one or more embodiments and/orfigures disclosed and described herein. In one embodiment, log data fromlogs 150(1)-(N) associated with clients 145(1)-(N) is received at MDRserver 105. Log anomaly detection engine 110 generates a contextassociated training dataset by splitting a string in a log (e.g., logs305(1) or 305(2)) as shown in FIGS. 3A and 3B) into multiple splitstrings (e.g., NLP string 405 as shown in FIG. 4 ). This process can beperformed for multiple logs simultaneously by log anomaly detectionengine 110. For example, once one or more logs are accessed, string keysin a given field are tokenized and split into separate words.

In one embodiment, associations of sub-strings (e.g., also called splitstrings) are generated and trained together (e.g., against each other)to generate a single vector (e.g., a dense vector) for each log, whichis then analyzed using one or more machine learning techniques (e.g.,clustering, classification, and the like). For example, in certainembodiments, a context association is generated between each splitstring a unique key that corresponds to the log. The context associationmaintains an association between a key and one or more words that havethe key. For instance, a path (e.g., a key for a file path value) issplit by slashes (e.g., source text 205 as shown in FIG. 2 ), which caninclude getting folder names and each token that is generated (e.g.,file path—C: Drive, file path—Windows Directory, and the like).

The foregoing tokenization is performed for all values (e.g., splittingon spaces, and the like, for a command like). In some embodiments, oncethe log data is received, log data is batched and read out to a pipeline(e.g., an ETL pipeline) that splits each log into individualized tokensconcatenated with a key (e.g., a unique key). Each token keeps aconcatenation of a key name and a split string in the value (e.g., thevalue from the split string can be “Windows”). This string can appear inmultiple different key values (e.g., K/V pairs with key/value such asfolderpath/windows, parentfolderpath/windows, and the like). Therefore,tokenizing the value splits the value into a list of strings (e.g., alist of unique strings). Each string in the list of strings isconcatenated with the key.

When analyzing forensic log data, strings (e.g., string values) thatcome out of the tokenization for a given K/V pair are different fromanother K/V pair if the key (e.g., the key name) is different. Loganomaly detection engine 110 obtains a bunch of strings per log andmaintains them in sets (e.g., an association of which log has whichstring). Therefore, each log line (or log) gets split into sub-stringsthat are then kept in batches by log anomaly detection engine 110 (e.g.,by log manager 115).

In one embodiment, log anomaly detection engine 110 generates aninput/output (I/O) string data batch that includes I/O string data foreach split string in the log. The I/O string data is generated bytraining each split string against every other split string in the log.In this example, the input is every string matched with every otherstring in the log. The output in this case includes a contextassociation that two or more tokens that exist(ed) in the same log, thusenabling the tokens and the split strings to be trained together. Forexample, if there are 5 split strings in the log, string 1 is trainedagainst strings 2, 3, 4, and 5, string 2 is trained against strings 1,3, 4, and 5, and the like (e.g., as shown with respect to trainingsamples 210(1)-(N) in FIG. 2 ).

Therefore, log anomaly detection engine 110 generates a full set ofinputs/outputs for each split string to every other split string in agiven log set that permits a log line that is tokenized into splitstring or sub-strings from the same log to be trained together. In someembodiments, I/O data from multiple logs is concatenated together andbatched before being trained. The output of the training, which issequential, is a list of split strings (e.g., a list of unique splitstrings). Therefore, the table of strings (e.g., the list of (unique)(split) strings) for model training includes only unique strings (e.g.,filepath/windows, filepath/myprogram, and the like).

In certain embodiments, the foregoing training (e.g., for acontext-based anomalous log data identification model) is performedaccording to one or more machine learning techniques using the I/Ostring data batch (that includes the list of unique strings) in thecontext associated training dataset. The table includes a mapping ofeach unique token to a dense vector that is being trained. The vectormaintains metadata that relates the vector to other split strings thathave been or are part of similar logs. The training tunes thecontext-based anomalous log data identification model to classify orcluster a (new) vector associated with a new string in a new log that isnot part of the plurality of logs as anomalous.

In some embodiments, log anomaly detection engine 110 (as shown in FIG.1B), generates a single dense vector for a log by accessing a list ofunique split strings and averaging several vectors that include at leastone vector for each unique split string in the list of unique splitstrings. The dense vector indicates a mapping of each unique splitstring in the list of unique split strings to the dense vector beingtrained. Therefore, instead of multiple vectors for a log, a singlevector is generated. The single dense vector can then fed into a machinelearning model trainer to generate the context-based anomalous log dataidentification model.

In other embodiments, the context-based anomalous log dataidentification model is also trained with additional I/O string datagenerated for each log that is being processed by log manager 115.Multiple logs can be processed sequentially and/or simultaneously.Examples of the composition of log data include, but are not limited to,process information (e.g., process names/hashes) associated withcomputing systems generating the log data (e.g., clients 145(1)-(N) asshown in FIG. 1A).

In certain embodiments, the context-based anomalous log dataidentification model is trained to perform cluster analysis based onwhether an executable that is part of the process information is a goodexecutable that is part of a bad path. The good executable and the badpath are pre-identified based at least on a classifier prior toperforming the cluster analysis. Log anomaly detection engine 110provides a single vector for a single log (as well as labels for eachvector). For example, all vectors can be represented as child processnames. Therefore, context-based anomalous log data identification modelcan identify particular anomalous processes (e.g., everything that iscommand.exe) and can do so as a cluster representation because a clusterof vectors are fairly close to each other.

In this manner, security analysts can start looking for anomalies fromthe center (e.g., malware, which in one embodiment, is an executable orthe way the executable is being used that tends to differ in log data).For example, in the case of a good executable and a bad path, if thepath name is randomized, the path name will be unique. However, becauselog anomaly detection engine 110 uses a single dense vector, the vectorthat the path name will be trained with will also be random. Therefore,a vector for a random folder name, for example, is likely to pull thelog vector away from all others (e.g., because the path is different).

In other embodiments, training the context-based anomalous log dataidentification model to perform cluster analysis is based at least on anumber of occurrences of a given process name/hash in the log (e.g., theclassifier can access existing logs that are labeled as benign ormalicious). For example, if the vectors are processed with a classifierwith the foregoing labels, the classifier can estimate whether thevector is malicious or benign.

In certain embodiments, and in reference to FIGS. 1A through 13 , loganomaly detection engine 110 generates from log data associated withlogs: (1) input data that includes a set of unique strings per log, (2)a word dictionary that maps a unique identifier for each field in eachlog to a list of sub-strings generated from a string value for eachfield in each log, and (3) a modified list of the logs comprising uniqueidentifier replacements for field values in each log. Word embeddingengine 125 then generates a set of word vectors by training one or moreword pairs in the set of unique strings for each log for co-occurrence.Field embedding engine 130 generates field vectors for each log thatincludes relationship information of sub-string components (e.g.,context-association) of the field vectors (from the set of word vectorsas shown in FIG. 6 ). Log embedding engine 135 then generates logvectors for each log from the field vectors. Finally, cluster analysisengine 140 performs cluster analysis to identify a part of the log datathat is anomalous based at least in part on the log vectors (e.g., logvector 510 as shown in FIG. 5 ).

As shown in FIG. 7 , and in some embodiments, log anomaly detectionengine 110 samples logs randomly (e.g., raw logs 605(1)-(N) randomsampling 705) and generates a hash of each log to utilize as a uniquelabel to permit disparately pre-processed versions of the same log to betracked together (e.g., log hashing 710). Next, JavaScript ObjectNotation (JSON) list structures (or other types of comparablestructures) are removed by converting a list of the sub-stringscomponents into a dictionary K/V pair. In addition, each filed in eachlog, an equivalent string value is converted to sub-strings and each newstring is compared to a list of bad strings to filter. Finally, theinput data, the word dictionary, and the modified list are generated.

In one embodiment, log anomaly detection engine 110 receives one or moreword pairs in the set of unique strings, generates a word vocabularythat is a mapping of the set of unique strings to one or more integerindexes, and configures configuring the one or more integer indexes tomap a unique string in the set of unique strings to a word vector of theset of word vectors.

In another embodiment, log anomaly detection engine 110 also trains oneor more word pairs in the set of unique strings for each log forco-occurrence by training each string in a log with every other stringin the log, and based on the training, generates the set of word vectorsthat commonly co-occur with each other in a similar area of vector spaceand saves the set of word vectors as a word embedding tensor. Loganomaly detection engine 110 also receives the word dictionary, the wordvocabulary, and the word embedding tensor and aggregates each wordvector in the set of word vectors for each field value.

In some embodiments, and in reference to at least FIGS. 11 and 12 , loganomaly detection engine 110 generates a field vocabulary that maps theunique identifier replacements for field values to an integer index ofthe one or more integer indexes to lookup a field vector in a fieldembedding and converts the list of sub-strings into a list of indexes tolookup in the word embedding tensor to identify corresponding wordvectors. In other embodiments, log anomaly detection engine 110 performsan embedding lookup operation on the word embedding tensor (e.g.,embedding lookup tensor 1130), receives a tensor that is a subset of theset of word vectors (e.g., world embedding tensor subset 1135), andaggregates the field vectors across the substring components into afield embedding tensor (e.g., accumulate (sum) 1140).

In one embodiment, and in reference to FIG. 13 , log anomaly detectionengine 110 receives the field embedding tensor, the field vocabulary,the log dictionary, and the modified list of the logs, applies a filterto the modified list, and aggregates one or more field vectors for eachlog to generate the log vectors and a log embedding tensor. In anotherembodiment, and in references to FIGS. 6 and 7 , log anomaly detectionengine 110 receives the modified list of the logs comprising reorganizedraw logs, the log dictionary, and the log embedding tensor and performsthe cluster analysis to determine a distance between each log vector ofthe log vectors and an average of the log vectors (e.g., clusterdetection 665). In this example, the cluster analysis includesclustering data in the reorganized raw logs by one or more processesassociated with one or more computing systems generating the log data.In certain embodiments, cluster analysis engine 140 identifies processclusters generated by the clustering that are anomalous based on atleast a count of a number of occurrences of a process name/hash anddetermining a distance between each process cluster generated by theclustering to identify one or more process clusters that are farthestaway from other process clusters.

Additional Embodiments

Log 2Vec: Using NLP to Extract Features and Perform Cluster Analysisfrom Logs

Disclosed herein are methods, systems, and processes for using naturallanguage processing (NLP) to extract features and perform clusteranalysis from logs and forensic records (referred to herein as simply“records”). It will be appreciated that a significant number of moderncompanies and organizations use or require the use of a SecurityOperations Center (SOC). A SOC includes security analysts, securitysoftware, computer hardware (e.g., to execute said security software)and other mechanisms to combat and respond to cybersecurity threats.

Examples of security software that can be provided by a SOC via physicaland/or virtual computing systems and networks includes, but is notlimited to: vulnerability management (VM) systems, incident detectionand response (IDR) systems, penetration testing systems, applicationsecurity systems (both static and dynamic), security orchestration andautomation (SOAR) systems, and the like. In certain cases, a SOC isconfigured to perform managed incident detection and response(MDR)—which involves a company outsourcing its cybersecurity response toa third party that provides the SOC and associated capabilities andservices.

In the role of a security analyst in a SOC, there are at least two keyaspects of monitoring client systems and networks for malicious activityand data indicative of such malicious activity—identifying knownmalicious behavior occurring in such computing environments (also calledthreat detection) and identifying undetected malicious activity withinsuch computing environment (also called threat hunting).

Threat hunting is an important aspect of monitoring and in some casesthe only means of finding new malicious behavior and advanced threats.Threat hunting is typically performed by retrieving a large amount offorensic data from a customer's computing environment and storing theretrieved forensic data (e.g., raw logs) in a database. The database isthen manipulated in various ways in an attempt to discover outliers andanomalies. These outliers and anomalies are then further investigatedfor potential malicious activity or behavior that can pose threats tothe customer's computing environment or cause other types of harm (e.g.,data exfiltration, hack attacks, loss of sensitive information, and thelike).

In a typical SOC, the primary technique for threat hunting is called“Stacking.” Stacking involves taking one or more fields in a database(e.g., the database discussed above) and counting the frequency ofoccurrence of unique values. These values are commonly arranged by countfrom smallest to largest. Addition investigations can be performed if asecurity analyst (hereinafter referred to as simply “analyst”) considersany particular value suspicious.

While stacking has been shown to be effective in identifying certainthreats, stacking suffers from several shortcomings: (1) stackingrequires an analyst to sift through large amounts of data per stack, (2)workload increases substantially with the number of stacks, (3)visibility can be limited to only the fields that are stacked on, (4)stacking assumes that attackers will have a small footprint in thecomputing environment, and (5) stacking is less likely to highlightunusual inter-field relationships.

Example of Optimized Log Anomaly Detection

In one embodiment, a modified word embedding (e.g., word2vec) techniqueis used to convert unique word strings present in logs into vectors(e.g., logs 150(1)-(N) as shown in FIG. 1 ). The co-occurrence of wordspresent in the same log is used to extract relationship informationbetween log strings, leading the log strings to be pushed to similarareas of vector space. The word vectors are then used to create orgenerate vectors for each log. Cluster analysis and statisticaltechniques are then used on these log vectors to identify anomalousitems.

It will be appreciated that while machine learning, word embedding, andstatistical learning has traditionally been used to extract informationand meaning from human language, the methods, systems, and processesdescribed herein change or modify such existing systems, methods,processes, models, and techniques to identify occurrences andrelationships between items in log data in information technology (IT)environments. Doing so, permits analysts to identify not onlylogs/records that are completely different from other logs, but alsopermits analysts to identify logs/records that are trying to pretend tobe legitimate system artifacts or using legitimate system artifacts inan unexpected or malicious manner. It will be appreciated that thesystems, methods, and processes disclosed herein also permit (andfacilitate the) identification of anomalies for executables/artifactswhere the cryptographic hash in unknown by anti-virus and hash scanningtools.

For example, in the event that a customer's computing environment hascustom in-house software that is unrecognized by anti-virus systems, innormal circumstances, it would not be possible for an analyst to tell ata glance whether a given artifact or executable is malicious. However,with training data (e.g., training data 925 as shown in FIG. 9 ), a loganomaly detection engine 110 can identify anomalous attributes or usageof the foregoing in-house software that might indicate, for example, anadvanced attacker.

In one embodiment, one or more machine learning models (referred tohereinafter as “models”) that perform Natural Language Processing (NLP)and information retrieval paradigms such as Word2Vec and Global Vectorsfor Word Representation (GloVe) are applied to extract information fromlog data (e.g., from data from logs 150(1)-(N) produced by clients145(1)-(N) as shown in FIG. 1 ). Training models with data (in anunsupervised manner) ensures that these models can learn relationshipsthat exist between different data values.

Word2Vec (e.g., provided by a word embedding engine 125 as shown in FIG.1 ) works by taking a large text corpus (e.g., the entire text ofWikipedia) and for every unique word, randomly initializing a vectorwith many dimensions (e.g., word vectors with 300 dimensions (orcomponents)). A cost function is then created based on the followingmodel: for a given string value for a center word at some point “x” in acorpus “C” (i.e., “C[x]”), the likelihood of a given string valueappearing next to the center word within a certain range is determined(e.g., if the window has a length of L=3, does the outer word appear inlocations “C[x−3]”, “C[x−2]”, “C[x−1]”, “C[x+1]”, “C[x+2]”, “C[x+3]”?).The cost function can be used to compare randomly initialized wordvectors and back propagation can be used to make incremental changes tothe individual dimensions (or components) such that the dot product ofthe center word vector and the window word vector is as high as possible(e.g., balanced out across the corpus by performing the same operationacross all center/window positions). Solving the foregoing problemacross the corpus produces a minimum (zero point) for the componentvalues of all word vectors.

In certain embodiments, log anomaly detection engine 110 implemented byan MDR server 105 performs at least five core steps and one or morecombinations thereof: (1) Pre-processing, (2) Word2Vec, (3) Field2Vec,(4) Log 2Vec, and (5) Cluster analysis. Each of these five steps are nowdiscussed separately in greater detail.

Example Log Pre-Processing for Anomaly Detection

In one embodiment, log pre-processing for anomaly detection is performedby a pre-processing engine 120. In this example, log pre-processinginvolves converting log data (e.g., from logs 150(1)-(N) into input datarequired by word embedding engine 125, a field embedding engine 130,and/or a log embedding engine 135 (as shown in FIG. 1 ). In oneembodiment, the input logs are in a consistent JavaScript ObjectNotation (JSON) format (but can be in any other suitable format such asYet Another Markup Language (YAML), Protocol Buffers, AXON, ConfigObj,OGDL, Further XML Exploration, and the like).

In some embodiments, pre-processing engine 120 performs at least thefollowing steps or operations: (1) randomly samples logs (optional), (2)generates a hash of each log to use as a unique label to use for arecord such that differently pre-processed version of the same log canbe tracked together, (3) removes JSON list structures by converting listcomponents into a dictionary key/value (KV) pair where the key is anequivalent index lookup of that item (e.g., “[0]” and where the value isthe item itself making it easier for the JSON to be filtered), (4)implements a filtering components that permits a user to decide whichfield components of the log are to be trained on based on a providedJSON filter (e.g., training filter 805 as shown in FIG. 8 ), providingan effective way of removing fields that would be bad for use in wordembeddings (e.g., Word2Vec), such as fields that are integers.

In step/operation (5), and in certain embodiments, for each field in thelog, the field's equivalent string value is broken down into substrings.This step has three sub-steps: (a) pattern matching is performed to findfiles, folders, and/or domains that contain spaces, and the spaces arereplaced with underscores, (b) the strings are split by slash and spacecharacters (e.g., as opposed to training entire unique paths againsteach other because some paths will have a random or varying foldernames, each occurrence of which would be treated as a unique word totrain against—by splitting the folders the parts of the paths that areconsistent can be reliably trained against), and (c) the field name iscombined with each unique data string.

In step/operation (6), and in some embodiments, each new string iscompared to a list of bad strings to filter (e.g., to remove customersensitive strings from training data such as users, hostnames, domains,and company name). In addition to anonymizing the data, this steppermits trained words to be customer agnostic, ensuring that data frommultiple customers (e.g., clients 145(1)-(N)) can be trained and thesame vector model can be applied to multiple customers. Instep/operation (7), and in other embodiments, three sets of data areoutputted by the pre-processing stage (e.g., by pre-processing engine120): (i) a set of Word2Vec input data that includes a set of uniquestrings per log, (ii) a dictionary mapping each unique field valueidentifier to (its) list of substrings, and (iii) a modified list oflogs where each field is replaced by a unique identifier for that value.In the final step/operation (8), the original log data is reorganizedinto a dictionary where each log can be looked up by the log's uniquehash label (e.g., as shown in FIG. 7 ).

Example Word Embedding(s) for Anomaly Detection

In one embodiment, word embedding engine 125 accesses or receives a setof training strings as input (e.g., training strings 625(1)-(N) as shownin FIG. 6 or trainingstrings.json 765 as shown in FIG. 9 ). Wordembedding engine 125 then builds a vocabulary (e.g., a mapping of uniquestrings to integer indexes to use for mapping the unique strings to aparticular vector). In this example, the vocabulary is output as a .jsonfile so that later components can easily look up the index forparticular words (e.g., word2idx.json as shown in FIG. 9 ).

Next, the training inputs and outputs are created (e.g., generateinputs/outputs 920 as shown in FIG. 9 ). In one embodiment and at thisstage, word embedding engine 125 is configured to perform variousoperations that differ from existing word2vec models. For example,existing word2vec models work from a single large corpus of text andgenerate training pairs by taking a given center word and a window thatis +/−a certain window size from the center word. The center word andthe window of the center word are then moved through the corpus and witheach step, training pairs are generated. The expected inputs are thecenter word for the given step and the expected outputs are the wordsthat exist in the window for the given step. The foregoing can beillustrated with an example sentence “that car is very good” with thewindow being +/−1 word from the center word:

-   -   [{“that”} “car”] “is” “very” “good”→C=“that”˜W=[“car”]→[“that”,        “car”]→[0, 1]    -   [“that” {“car”} “is”] “very” “good”→C=“car”˜W=[“that”,        “is”]→[“car”, “that”], [“car”, “is”]→[1, 0] [1, 2]    -   “that” [“car” {“is”} “very”] “good”→C=“is”˜W=[“car”,        “very”]→[“is”, “car”], [“is”, “very”]→[2, 1] [2, 3]    -   “that” “car” [“is” {“very”} “good”]→C=“very”˜W=[“is”,        “good”]→[“very”, “is”], [“very”, “good”]→[3, 2] [3, 4]    -   “that” “car” “is” [“very” {“good”}]→C=“good”˜W=[“very”]→[“good”,        “very”]→[4, 3]

In some embodiments, after the training pairs are generated, a set ofrandomly initialized high dimensional vectors are created for eachunique word. Then, the expected input and output data is fed into anoptimizer with randomly generated weights and biases. Over the course oftraining, the optimizer adjusts the vectors such that the vectors becomethe closest representation of the expected input and output data.

In other embodiments, with the modified word embedding techniquedisclosed herein, instead of using a sliding window to generate trainingpairs, word embedding engine 125 groups training data (e.g., trainingdata 925 as shown in FIG. 9 ) into sets. Instead of using a slidingwindow, word embedding engine 125 produces training pairs forco-occurrence of any two words in that (given) set. Implementing theforegoing technique ensures that the order of strings does not matterand relationships between every string in a log is trained. For example(I being “input” and O being “output”):

-   -   I=“that”˜O=[“car”,“is”,“very”,“good”]→[“that”, “car”], [“that”,        “is”], [“that”, “very”], [“that”, “good”]    -   I=“car”˜O=[“that”, “is”, “very”, “good”]→[“car”, “that”],        [“car”, “is”], [“car”, “very], [“car”, “good”]    -   I=“is”˜O=[“that”, “car”, “very”, “good”]→[“is”, “that”], [“is”,        “car”], [“is”, “very”], [“is”, “good”]    -   I=“very”˜O=[“that”, “car”, “is”, “good”]→[“that”, “car”],        [“very”, “car”], [“very”, “is”], [“very”, “good”]    -   I=“good”˜O=[“that”, “car”, “is”, “very”]→[“good”, “that”],        [“good”, “car”], [“good”, “is”], [“good”, “very”]    -   I=0˜O=[1,2,3,4]→[0,1] [0,2] [0,3] [0,4]    -   I=1˜O=[0,2,3,4]→[1,0] [1,2] [1,3] [1,4]    -   I=2˜O=[0,1,3,4]→[2,0] [2,1] [2,3] [2,4]    -   I=3˜O=[0,1,2,4]→[3,0] [3,1] [3,2] [3,4]    -   I=4˜O=[0,1,2,3]→[4,0] [4,1] [4,2] [4,3]

In this example, a set of randomized dense vectors, an optimizer,weights, and biases are used. The expected input/output data is fed intothe optimizer and the result is a set of trained vectors, where vectorsthat commonly co-occur with each other occupy a similar area of vectorspace. In one embodiment, a machine learning framework such asTensorFlow is used for vector manipulation. In this example, the NoiseContrastive Estimation learning technique and the Adam optimizer is alsoused. It should be noted that in other embodiments, alternate or othermachine learning frameworks, learning techniques, and optimizers arealso contemplated. The final output of the word2vec or word embeddingcomponent is a TensorFlow model (or a comparable model generated fromanother machine learning framework) with a word embedding tensor (e.g.,a word embedding that is a list of word vectors by index and iseffectively a matrix).

Example Field Embedding(s) for Anomaly Detection

In one embodiment, the inputs for field embedding engine 130 (e.g., thefield2vec component) is a dictionary set that maps identifiers forunique fields to a list of substrings for that field, the word to indexvocabulary, and the TensorFlow word embedding. In this example, the wordvectors for each unique field are summed up (or added) and a fieldvector is obtained that contains the relationship information ofsubstring components. First, a vocabulary is generated by log anomalydetection engine 110 that maps the identifiers for unique fields to aninteger index that can be used to look up (or reference) the fieldvector in the field embedding.

Next, the dictionary that maps unique field value identifiers to thelist of substrings is accessed and the list of substrings is convertedinto a list of indexes (e.g., list of indexes 1125 as shown in FIG. 11 )to look up in the word embedding (e.g., to identify corresponding wordvectors). In this example, log anomaly detection engine 110 uses theforegoing lists as word embedding lookups (e.g., as shown in FIG. 11 ).

In some embodiments, list of indexes 1125 uses a machine learningframework vector (e.g., as an embedding lookup vector). Also, because anembedding lookup vector exists for each unique field identifier, anembedding lookup tensor is created (e.g., a list of embedding lookupvectors). In this example, the vectors in the tensor are arranged basedon the index of their corresponding unique field identifiers.

Maintaining lookups in tensor form (e.g., embedding lookup tensor 1130and embedding lookup tensor (subset) 1135) optimizes look ups for easierand faster manipulation with machine learning frameworks such asTensorFlow. However, one shortcoming that exists with the foregoingmechanism is that there is no consistency to the length of the embeddinglookup vectors. In a framework such as TensorFlow, tensors withinconsistent shape are called “ragged” tensors on which operationscannot be performed.

However, in certain embodiments, one or more methods can be used toconvert a ragged tensor to a regular tensor (e.g., using a techniquecalled masking). However, in this example, since the indexes in thelookup vectors are going to be used to look up word embeddings, aslightly different approach is used. In one embodiment, each embeddinglookup vector is configured to be the same length of the longest lookupvector and the shorter lookup vector is padded with an index to a “zerolookup” (e.g., a lookup to a zero vector—a vector of the same lengthwith values set to zero). Therefore, when the sum of the word vectorsoutput by the word embedding lookup is calculated, the zero vectors donot affect the sum. To configure the foregoing, in one embodiment, thezero vector is added to the end of the word embedding.

In some embodiments, to create the field vectors, TensorFlow's map_fnoperation is used. The foregoing operation receives a tensor and asub-operation as input, breaks up the inputted tensor into sub-tensorswith shapes of the lowest dimension, performs the sub-operation on each,and then re-combines the results. In the case of the embedding lookuptensor, the foregoing function splits the tensor into individualembedding lookup vector and performs an operation on each, and thenre-combines the results. For each embedding lookup vector, the followingoperations or steps are executed: (a) performing an embedding lookup onthe word embedding, (b) receiving a tensor that is the subset of wordvectors, (c) summing (or adding) the tensor vectors across thecomponents, and (d) optionally, normalizing (average) the field vector.

In some embodiments, the output of the foregoing operation(s) is a fieldvector (e.g., field vector 650(1)) (e.g., for each loop of the map_fnfunction, a field vector is generated). Then, at the end of the loop,the generated field vectors are automatically combined into a tensor,which in this example, is the field embedding tensor. In this example,the field embedding tensor is the final output of the Field2Veccomponent (e.g., implemented by field embedding engine 130 as shown inFIGS. 1A and 11 ).

Example Log Embedding(s) for Anomaly Detection

In one embodiment, the inputs for log embedding engine 135 are thedictionary field embedding tensor (e.g., log 2idx.json 1315 as shown inFIG. 13 ), the vocabulary/dictionary that maps identifiers for uniquefields to an integer index, and the dictionary that maps log hashes to amodified log structure (e.g., where field values are uniqueidentifiers).

In certain embodiments, the purpose of log embedding engine 135 is togenerate log vectors for each log from the sum of the given log'scorresponding field vectors (e.g., as shown in FIG. 13 ). In someexamples, the implementation for summing field vectors and collating logvectors is similar to the field embedding engine 130. However, in someembodiments, before the summing and collating is performed, logembedding engine 135 applies a filter to the modified log's input tofurther filter fields. This permits users to train on the word vectorsfor a broad variety of fields (less filtered). Then, at a later stage,user(s) can remove the vector component of particular fields whenperforming cluster analysis.

Example Cluster Analysis for Anomaly Detection

In one embodiment, the inputs for cluster analysis engine 140 are thereorganized raw logs, the dictionary/vocabulary that maps log hashes tointeger indexes, and the log embedding tensor. In this example, usingthe data in the raw logs, cluster analysis engine 140 splits the logvectors by unique process names. These log vector subsets can be treatedas their own tensors and statistical analysis can be performed on the(log) items. In addition, the map_fn function (or any similar orcomparable function of another machine learning framework other thanTensorFlow) can be used to sum (or add) the log vectors in a subset andthen an average log can be created for that (given) subset by dividingthe log by the scalar count of the logs in the subset.

Once an average log is obtained, cluster analysis engine 140 accessesthe log subset for a given process and calculates either the cosine orEuclidean distance between each log vector and the average log vector.For anomaly detection, and in some embodiments, logs that are farthestfrom the center of the cluster are of particular interest. For example,because many of the process strings for the executable will (likely) bethe same, it is also likely that the vector for the process log will beclose to the given cluster (e.g., the cluster under analysis). Toconsider and cover the foregoing cases, and in other embodiments,cluster analysis engine 140 uses the set of distances to calculate thestandard deviation of the distance from the cluster. Consequently, thetensor can be accessed for (all) log vectors and the distance between(all) logs and the (given) average log can be calculated. The resultscan then be filtered to the vectors that are within a pre-determinednumber of standard deviations of distance.

In certain embodiments, the foregoing cluster analysis process can beperformed with a MD5 hash field (e.g., it does not have to be performedwith just the process name field, as described above). In addition, andin some embodiments, the most unusual process clusters in a givencomputing environment can be identified by considering at least twoadditional factors. First, the number of occurrences for a given processName/Hash can be determined. Second, the distances between all clusterscan be determined and the clusters farthest away from the rest can beidentified. Although the processing for the second factor can becomputationally expensive if the number of clusters significantlyincreases, other computationally efficient processes such as RandomForest can be implemented to determine ‘how’ abnormal a given clusteris.

Other Example Embodiments

It will be appreciated that the methods, systems, and processesdisclosed herein permit managed detection and response analysts in a SOCto visualize clusters of logs to visually identify anomalies and/orpermit computation-based identification of anomalies in clusters. Thetime required for threat hunting is thus likely reduced, leading topotentially faster hunts and the ‘most’ suspicious log data can bedisplayed to the analyst in a timely manner.

The methods, systems, and processes disclosed herein use naturallanguage processing (NLP) to enable MDR server 105 to detect and analyzelog and forensic data for anomalies and visualize the results of theanalysis (e.g., as shown in FIG. 1B).

In one embodiment, the script used to detect, analyze and visualizeanomalies in log data is log agnostic. Any one or more of a number oflog sources can be part of logs 305(1)-(n). In some embodiments, whenthe data is summed or aggregated, the data is normalized. This isbecause the final position in vector space of a log can be significantlyaffected by just how many unique strings they have. For example, ifthere are two sets of similar winword logs, and one set has 15 uniquestrings and another set has 20 unique strings, even if those strings arethe same, the two sets can form into different clusters. However, incertain embodiments, if instead of a sum, an average is performed, thetwo sets can become one in vector space. This is particularly useful inthe instant situation because log anomaly and detection is moreinterested in strange or unusual relationships between strings, ratherthan how many strings a given log has.

In other embodiments, other word vectorization models such as GlobalVectors for Word Representation (GloVe) are contemplated. In thisexample, instead of sending individual training pairs (or batches oftraining pairs) to an optimizer (e.g., part of word embedding engine125), a sizeable part of preprocessing generates a large co-occurrencematrix. Providing this co-occurrence matrix to the optimizer can be moreefficient.

In some embodiments, various hyperparameters that can affect the successof information retrieval are contemplated (e.g., the number ofdimensions for dense vectors). In a typical NLP use case, therecommended size is around 200-300 dimensions (e.g., because ofdiminishing returns after 400 dimensions). In one embodiment, a testingmetric is implemented that permits the determination of success oftraining against a set of logs. In this example, this training ‘success’can be measured with just the vectors (e.g., by using linguisticexamples of relationships between words-vector differences betweendifferent tenses of a verb, and the like).

Therefore, the question of how well certain parameters positively ornegatively affect vector relationships can be tested (and determined) bylooking at and analyzing the variation of distance for theserelationships. Similar examples can be set up for logs (e.g.,determining vector difference between 32-bit processes and 64-bitprocesses, and the like). Alternatively, the success of parametercombinations can also be tested by feeding vectors into a log classifierand determining how the success of the classifier changes with vectorsthat have been generated with different parameter sets.

In one embodiment, field embedding engine 130 generates custom weightsfor particular JSON fields. For example, the primary influence fortraining can be two particular fields while the rest of the fields canbe configured to have a minimum amount of training (e.g., prioritizingprocessName, md5 and cmdLine fields over other fields in process logs).In other embodiments, log vectors are fed into more complex machinelearning tasks such as classifiers or Recurrent Neural Networks.

Examples of Natural Language Processing (NLP) for Log Anomaly Detection

In certain embodiments, (a) Natural Language Processing (NLP) includesone or more techniques for extracting data from natural language, (b)Vector means a point in space represented by numbers (e.g., x=1, y=2,z=3), (c) Embedding is a ‘Vector,’ (d) Tensor is a set of points inspace that can be represented by a matrix, (e) Word2Vec is an algorithmfor turning words into vectors, (f) Cluster is a collection of itemsthat have similar properties, (g) Standard Deviation is a statisticalmethod of measuring the size of a distribution/cluster, (h) TensorFlowis a machine learning framework, (i) PCA is Principal ComponentAnalysis, and (j) T-SNE is t-Distributed Stochastic Neighbor Embedding.

In cybersecurity computing environments, threat detection involves thefollowing characteristics: the source for threat detection is previousmalicious activity, the activity follows a known pattern, rules can bebuilt (e.g., alerts can be generated whenever activity occurs), andalerts are direct leads for investigations (e.g., by a SOC analyst). Onthe contrary, threat hunting involves the following characteristics:there is no source for threat hunting, activity follows an unknownpattern, rules cannot be built, computing and networking environmentsmust be searched for new and unknown behaviors, and hunting attempts togenerate investigation leads for SOC analysts from computing andnetworking environment data.

Threat hunting (or simply ‘hunting’) involves accessing or retrievingavailable data in a computing/networking environment (e.g., collatinglog data for a time period, retrieving forensic data from endpoints, andthe like), manipulating the data in some manner (e.g., identifyinglocations commonly associated with malicious activity, identifyingunique items (e.g., stacking), and the like), identifying some sort ofanomaly (e.g., manually), and investigating said anomalies (e.g., timeconsuming, reserved only for high confidence items).

In certain embodiments, hunting includes collecting forensic artifactsfrom endpoints and mining log data for report CSVs (comma-separatedvalues). The forensic records are stored in a database (e.g., database2510 as shown in FIG. 25 ) and the database is queried with stackingqueries. The stacks are then examined. For example, in some embodiments,the (computing and/or networking) environment data gathered by MDRserver 105 is the output of various forensic tasks from variouscomputing assets (e.g., physical computing devices and/or virtualcomputing devices). The data is then parsed and stored in the database.A field range is chosen and the unique values are arranged for that(given) field by frequency of occurrence (e.g., to focus attention onfield values that are uncommon in the environment because maliciousattackers can often leave a small footprint).

The benefits of stacking include the ability to target fields mostlikely to identify or detect malicious behavior, the ability toprioritize records with unique values, and the ability to spot acomplete irregularity (e.g., a randomized string). The negatives ofstacking can include too much data, an increase in workload with thenumber of stacks, limited visibility only to stacks, and the need for anassumption that attackers will have a small imprint in the environment.For example, attackers filling fields with bad data or typos can becaught by stacks (e.g., particularly in the subset of maliciousprocesses where every field in unique). Therefore, stacking is good forlooking at known fields that commonly reflect anomalous behavior, givingitems a priority (unique values are more valuable than common values),and spotting malicious attackers who fill in fields with easy to spotbad data and/or make typographical errors.

In a first example, an attacker can decide that they want to disguise amalicious file called bad.exe as Chrome.exe. The attacker changes nameto Chrome.exe and replaces the Chrome.exe in C:\Users\(useraccount)\AppData\Local\Google\Application with no command linearguments, a salted hash, and falsified metadata (e.g., company name).Judging such disguised malicious files becomes difficult the less wellknown (and bland) the process that is being imitated (e.g., it would bereally difficult to judge an attacker pretending to be an obscure thirdparty application or in-house custom software). In a second example, anattacker can also use PowerShell in a manner than anti-virus does notdetect by renaming PowerShell to Chrome.exe where CmdLine arguments are<<regular chrome args>>+<<malicious powershell>>. In this example, thehash is valid and the metadata is already valid.

In a third example, an attacker can disguise a malicious file calledbad.exe as Winword.exe. In this example, the bad file names itselfWinword.exe (changes metadata information to reflect Winword), replaceswinword.exe executable (e.g., in C″\Program Files (x86)\MicrosoftOffice\root\Office16), and changes command line arguments to lookwinword-like (e.g., opening particular documents). The hash is salted.However, in this example, the relationship between fields and hash isanomalous and there is a non-existent signing chain or incorrect signingchain (e.g., signed by Bad.co).

In a fourth example, an attacker disguises powershell.exe asWinword.exe. In this example, the attacker changes the name toWinword.exe (no need to change metadata) and replaces the winword.exeexecutable in C″\Program Files (x86)\Microsoft Office\root\Office16.There are no command line arguments (“live” shell) and the hash isunchanged (e.g., will not appear on a Reversing Lab unknown). However,in this example, the relationship between the winword process name thepowershell fields is anomalous.

One problem with respect to detecting anomalies in log data is that itis relatively easy to determine that a record is unique overall butchallenging to arrange data in a manner that prioritizes records withcommon strings and one or two unique strings. In one embodiment, loganomaly detection engine 110 configures word embedding engine 125 togenerate a machine learning algorithm that can be used for NLP. Wordembedding engine 125 transforms words into vectors where the vectorscapture word relationships. In this manner, word embedding engine 125identifies synonyms between words. In some embodiments, the machinelearning algorithm of word embedding engine 125 is trained over a largetext (e.g., Wikipedia) and turns words into “dense” vectors (e.g., with100 to 500 dimensions). Therefore, words that commonly occur next toeach other in the text will be closer to each other in vector space.

FIG. 1A is a block diagram 100A of a managed detection and response(MDR) server and FIG. 1B is a block diagram 100B of a MDR server thatimplements a log anomaly detection engine, a user, hostname, process(UHP) engine, a Markovian analysis engine, and a security operationsengine, according to certain embodiments. MDR server 105 can be any typeof computing device (e.g., a physical computing device or server with aprocessor and memory) and implements log anomaly detection engine 110.

Log anomaly detection engine 110 includes log manager 115 (e.g., forreceiving, organizing, and managing log data from raw logs or othercomparable forensic records), pre-processing engine for performing logpre-processing (as discussed above), word embedding engine 125 (e.g., toimplement Word2Vec), field embedding engine 130 (e.g., to implementField2Vec), log embedding engine 135 (e.g., to implement Log 2Vec), andcluster analysis engine 140 (e.g., to perform cluster analysis after acombination of the pre-processing, word2vec, field2vec, and log 2vecprocesses discussed above). MDR server 105 is communicatively coupled toclients 145(1)-(N) via network 155 (which can be any type of network orinterconnection). Clients 145(1)-(N) each include one or more logs(e.g., logs 150(1)-(10) from client 145(1), logs 150(11)-(30) fromclient 145(2), and the like). The logs are retrieved from clients145(1)-(N) by log manager 115 or sent to log manager 115 by clients145(1)-(N).

MDR server 105 includes a processor 160 and a memory 165. As shown inFIG. 1B, memory 165 implements at least log anomaly detection engine110, and in addition, or optionally, a user, hostname, process (UHP)timeliner engine 170, a Markovian analysis engine 175, and a securityoperations engine 180. Log anomaly detection engine 110, UHP timelinerengine 170, and Markovian analysis engine 175, either alone or incombination, enable MDR server 105 to perform optimized log anomalydetection, analysis, and visualization.

FIG. 2 is a block diagram 200 of a word embedding model for naturallanguage processing (NLP), according to one embodiment. Word2Vec (whichis an example of a word embedding model), goes through each word of textand uses the word as the center word (e.g., a certain window sizing of+/−3 words from the center word can be used). Each word vector israndomly initialized at the start and each center/window wordco-occurrence is a Loss function (e.g., the likelihood of“quick”→“fox”). Word embedding engine 125 can minimize the loss problemacross words and training samples, although there are certain sets ofvalues for word vectors that cannot be further minimized.

FIG. 3A is a block diagram 300A of a log shown as continuous text andFIG. 3B is a block diagram 300B of a log with a window size smaller thanthe log, according to some embodiments. As shown in FIG. 3A, one problemis that if logs are treated as continuous text, inter-log relationshipswill be trained. As shown in FIG. 3B, another problem is that if awindow size is smaller than the log, relationships can be missed. Logscan however be turned into a large continuous text of strings.Unfortunately, doing so results in a center word at the end of a logthat will have a window that overlaps into the next log, poisoningrelationships. In addition, with a small window, relationships betweenfront and end strings of the log will not be trained. In this manner,Word2Vec can be applied to logs.

Example of Modified Word2Vec Output

In certain embodiments, Word2Vec is modified by word embedding engine125 by training every string in a lot with every other string in thatlog. In this example, applying Word2Vec to logs involves a minormodification to word2vec. In one embodiment, the window for any givencenter word is the entire log (or sentence). FIG. 4 is a block diagram400 of a modified word embedding model, according to one embodiment. NLPstring 405 illustrates an example of center/window word sets for “thatcar is very good.” FIG. 4 also illustrates the example contents of log305(n) and a modified word embedding 410.

FIG. 5 is a block diagram 500 of summed word vectors, according to oneembodiment. Word vectors 505(1)-(3) when summed, results in log vector510. Word2Vec produces a set of vectors for each string (e.g.,“Winword.exe”)—however, this is not very useful for threat hunting. Onthe other hand, and in certain embodiments, a vector version of logs isgenerated by listing unique strings in a log, summing all vector valuesfor strings, and normalizing the log vectors. Because vectorrepresentation of logs are desired, vectors for each unique string in alog set are accessed by log anomaly detection engine 110 and for eachstring in a log—its word vector representation is generated/derived.Then, the word vectors are summed to the log vector (e.g., log vector510 as shown in FIG. 5 ).

Example of Log Vector Clustering

Multiple instances of vectors that are similar to each other tend toform into clusters. In one embodiment, log anomaly detection engine 110performs log vector clustering. Logs that are similar will grouptogether and logs for the same processes can (nearly always) contain thesame strings. Strings in the log should have trained relationships andlog clusters will form in vector space. Anomalous logs will be thefarthest outliers for a given cluster.

Example of Clustering Statistics

In one embodiment, logs are filtered to a subset (e.g., logs whereprocessName=“winword.exe”. The average “winword.exe” log is determinedand the average log is represents the center of the cluster. Thedistances of “winword.exe” logs from average is determined (e.g., thedistance should be a Normal distribution). The standard deviation isthen determined and the “winword.exe” logs starting with those that arethe furthest away (e.g., the anomalies) are listed.

Example Benefits of Log Vectors for Threat Hunting

It will be appreciated that logs for the same processes should nearlyalways contain the same strings. Since these strings have co-occurred,they should have similar components. For example, the log vector for“winword.exe” is pushed to a very “winword” part of vector space. If alog contains an in appropriate non-winword string, then the log vectorwill be pulled away from the rest. Consequently, anomalous logs will be“X” number of standard deviations away from the center of the cluster.

Example Diagrams of Log Anomaly Detection Computing Systems

FIG. 6 is a block diagram 600 of a log anomaly detection system,according to one embodiment. The log anomaly detection system includesat least raw logs 605(1)-(N), log preparation 610, modified logs615(1)-(N) (e.g., after the pre-processing), unique fields 620(1)-(N),training strings 625(1)-(N), word2vec 630, word vocabulary 635, wordvectors 505(1)-(N), field2vec 640, field vocabulary 645, field vectors650(1)-(N), log 2vec 655, log vocabulary 660, log vectors 510(1)-(N) andcluster detection 665.

FIG. 7 is a block diagram 700 of pre-processed logs and outputs ofmodified logs, field strings, and training strings, according to oneembodiment. In one embodiment, FIG. 7 illustrates a log pre-processingscript (e.g., (1) filter JSON 720 (e.g., using a training filter 805 asshown in FIG. 8 ), (2) extract strings 725, and (3) filter bad strings730 as shown in FIG. 7 ). For example, string values are broken down byspaces and slashes. For a simple field like processName, there is onlyone string. For complex fields like path or cmdLine, there are a list ofstrings (e.g., “executablePath←C:”, “executablePath←Windows”,“executablePath←findstr.exe”). Complex regex is used to capturefolders/files with spaces and the spaces are replaced with underscores.

Further, as shown in FIG. 7 , the log anomaly detection systemconstructs mod-logs 740 (e.g., (A) modlogs.json), constructs uniquefield sets 750 (e.g., (B) fieldstrings.json), and constructs trainingstring sets 760 (e.g., (C) trainingstrings.json)—outputs of modlogs,fieldstrings, and output training strings, respectively.

FIG. 9 is a block diagram 900 of a word embedding implementation (e.g.,Word2Vec script 905), according to one embodiment. FIG. 10 is a blockdiagram 1000 of word vectors, according to another embodiment. FIG. 11is a block diagram 1100 of a field vector implementation (e.g.,Field2Vec script 1105), according to some embodiments. FIG. 12 is ablock diagram 1200 of field vectors, according to other embodiments.FIG. 13 is a block diagram 1300 of a log vector implementation,according to certain embodiments.

Example Processes for Log Anomaly Detection (Log 2Vec)

FIG. 14 is a flowchart 1400 of a process for detecting anomalous logdata, according to one embodiment. The process begins at 1405 byaccessing log data, and at 1410, performs log pre-processing. At 1415,the process generates word embeddings. At 1420, the process generatesfield embeddings. At 1425, the process generates log embeddings. At1430, the process performs cluster analysis and ends at 1435 bydetecting anomalous log data. In some embodiments, the process of FIG.14 can be performed by pre-processing engine 120 based on log datareceived from (and by) log manager 115.

FIG. 15 is a flowchart 1500 of a process for reorganizing original logdata, according to one embodiment. The process begins at 1505 byrandomly sampling logs and at 1510, creates a hash of each log to use asa unique label. At 1515, the process removes JSON list structures and at1520 identifies field components of the log to be trained. At 1525, theprocess generates substrings from the string value for each log, and at1530, compares each new string to a list of bad strings to filter. At1535, the process generates word embedding input data, dictionarymapping, and a modified list of logs, and ends at 1540 by reorganizingthe original log data into a dictionary (e.g., a frame in which sometraining data admits a sparse representation). In some embodiments, theprocess of FIG. 15 can be performed by pre-processing engine 120 incombination with word embedding engine 125.

FIG. 16 is a flowchart 1600 of a process for generating a model with aword embedding tensor, according to one embodiment. The process beginsat 1605 by receiving a set of training strings as input, and at 1610,builds a word vocabulary. At 1615, the process creates training inputsand outputs, and at 1620, accesses training data grouped into sets. At1625, the process generates training pairs for co-occurrence, and at1630, feeds expected input/output data into an optimizer. At 1635, theprocess receives a set of trained co-occurrence vectors and ends at 1640by generating a model (e.g., a machine learning model) with a wordembedding tensor.

FIG. 17 is a flowchart 1700 of a process for combining field vectorsinto a field embedding tensor, according to one embodiment. The processbegins at 1705 by accessing a dictionary set, a word to index embedding,and (the) word embedding tensor (e.g., from step 1640 in FIG. 16 ). At1710, the process creates a vocabulary that maps identifiers for uniquefields to an integer index, and at 1715, generates lookups in tensorform. At 1720, the process pads shorter lookup vectors with an index tozero lookup. At 1725, the process performs a map_fn operation (e.g.,shown in dotted lines “for each field” in FIG. 11 ), and at 1730,processes each embedding lookup vector. At 1735, the process generatesfield vector(s), and ends at 1740 by combining field vectors into afield embedding tensor. In some embodiments, the process of FIG. 17 canbe performed by field embedding engine 130.

FIG. 18 is a flowchart 1800 of a process for identifying anomalies inlog data, according to one embodiment. The process begins at 1805 byreceiving reorganized raw logs, a dictionary/vocabulary, and a logembedding tensor, and at 1810, splits the log vectors by unique processnames. At 1815, the process sums log vectors to create an average logfor the subset, and at 1820, accounts for (potential) renamedexecutable(s) from malicious attacker(s). At 1825, the processdetermines the distance between each log vector and the average logvector, and ends at 1830 by filtering results to vectors that are withinpredetermined standard deviations. In some embodiments, the process ofFIG. 18 can be performed by log embedding engine 135.

In addition to detecting anomalies in logs (e.g., using log anomalydetection engine 110), MDR server 105 also implements UHP timelinerengine 170 (e.g., to visualize anomalies in log data based on atimeline) and Markovian analysis engine 175 (e.g., to analyze anomaliesin log data), as shown in FIG. 1B.

Example of Visualizing Anomalies in Logs (UHP-Timeline Visualization)

One method of distinguishing regular activity from that of a maliciousattacker is to determine whether such activity is occurring outside of acompany's regular 9 am to 5 pm Monday through Friday working schedule.However, for a SOC analyst, there exists no methodology to easilynavigate and identify irregular activity outside of regular workinghours. Existing methodologies involve filtering log data to particularhours. However, this approach does not give the SOC analyst a sense ofwhat regular activity in the environment looks like—which limits theirability to identify genuine irregularities that can be indicative ofmalicious behavior.

FIG. 19A is a block diagram 1900A of a user interface of a UHP-TimelineVisualizer, according to one embodiment. In this example, a UHP-timelinevisualization interface 1905 is part of a UI application that permitsSOC analysts to classify logs according to a hash of values of acollection of fields, and display timeseries data for all such hashesacross a given computing environment in a manner that optimizes theability to visualize and search activity going on (or taking place)after hours.

For each process in a log, UHP timeliner engine 170 takes the User,Hostname, Process Path, Process MD5, and Process CommandLine and hashesthe combination of the foregoing values to generate a hash thatrepresents a particular activity in a given computing environment.Unique activities can often occur multiple times in a computingenvironment over the course of a day, and the timeseries data can beextracted. UHP timeliner engine 170 takes the timeseries data for hashesand displays them on top of each other on a particular timescale.

In certain embodiments, UHP-timeline visualization interface 1905provides the ability to drag and select log instances of activity, theability to filter and query the hashes to limit the quantities of data,and the ability to tag and whitelist particular hashes/activity asexpected. UHP-timeline visualization interface 1905 works with any JSONor key/value pair log type and the selection of fields that get hashescan be customized. In addition to the selection of a time period todisplay data from, UHP-timeline visualization interface 1905 permitsactivity of several days to be overlaid on each other on a 24-hourreview, or alternatively, permits activity over several weeks to beoverlaid on a 7-day Monday through Sunday timeline.

Currently, there exists no mechanism that permits a SOC analyst tovisualize and interact with a timeseries graph of log activity in agiven computing and/or networking environment. In addition, UHP-timelinevisualization interface 1905 is enabled to analyze time series data forhashes. In existing implementations, when considering UHP hashes, theonly timestamps stored per hash are “first_seen” and “last_seen”timestamps. This means that the same UHP activity could occur at a veryunusual time and would be missed. For example, if a system administratorstarts a remote desktop executable from an asset—that could be part oftheir work activities during the day—but would be unusual if theactivity occurred at 2 am on a Sunday.

UHP-timeline visualization interface 1905 includes at least a fieldordering 1910 with a source asset 1915, a source user 1920, adestination user 1925, a result 1930, and a destination asset 1935.UHP-timeline visualization interface 1905 also includes UHP hashes1940(1)-(N) and time series hits for UHP hashes 1940(1)-(N) 1945(1)-(N).As shown in FIG. 19A, and in certain embodiments, the black rectangularblocks within the wide dotted lines ( - - - ) (in the center between6:00 AM and 6:00 PM) can signal normal activity where as the blackrectangular blocks within the small dotted lines (on the left and on thebottom ( . . . )) (e.g., between midnight and 6:00 AM and for a whole24-hour period) can indicate anomalous activity and can providevisualization of such anomalous activity (e.g., to a SOC analyst).

In one embodiment, a user configures a set of logs to be viewed inUHP-timeline visualization interface 1905 and which key/value pairs ofthat log set to generate hashes for. The user can also select a limitedwindow for storage of the hashes and their equivalent timeseries dataand the granularity (e.g., hour, minute, second, millisecond) of thetimeseries data. Upon configuring UHP-timeline visualization interface1905, the log data in storage is accessed by UHP timeliner engine 170 toidentify and retrieve each unique hash, along with the hash's key/valuepairs and stores them in a database for later retrieval. Timeseries datais added to the database as the log data is being processed. Once thedatabase is complete, the hashes are used in the realtime loggingpipeline to track new instances of each hash and add to the timeseriesdata in the database. In addition, the database is permitted to age out(e.g., timeseries data that goes beyond the limit of configured andagreed retention). In this manner, live and realtime timeseries data isconstantly maintained (and refreshed) in the database (e.g., database2510) for visualization (and other) purposes in UHP-timelinevisualization interface 1905.

In certain embodiments, APIs are provided for downloading pre-bakedvisualization data to browser clients (e.g., not the timeseries dataitself) to minimize what would otherwise involve a large transfer ofdata.

In some embodiments, the UI-web application that is generated by UHPtimeliner engine 170 and provides UHP-timeline visualization interface1905 is configured as follows: the UI features a component that is akinto a combination of a spreadsheet and a music/audio editor. There is aportion of the UI dedicated to a list of all unique UHP hashes andanother portion of the UI dedicated to a timeline of activity in theform of a series of dashes for each hash (e.g., the black rectangularblocks illustrated in FIG. 19A, and discussed above). In addition, theUI includes UI components for ordering, sorting, and filtering thehashes by each of the fields that are hashed. Also included can becomponents to select a time range for data a SOC analyst wishes to view,including the option to overlay that time range onto a 24-hour period,Monday-Sunday period, or even a calendar-like monthly period. In someexamples, the visualizer (e.g., the UI, UI application, or UHP-timelinevisualization interface 1905) includes zoom mechanisms to permit a SOCanalyst to dig into specific behaviors. Timeline items can be selectedon click or by dragging a selection box over a few items. In certainembodiments, with a selection of log instances, the UI application caneither link to a log search page or provide a pop up log display windowwhere a SOC analyst can view the selected items in depth.

In one embodiment, a user can configure alerting for activity based on aspecific field (e.g., activity for a particular user or process if itoccurs during a particular time span in a 24 hour period or if it occursduring the weekend). Such alerts can be whitelisted for specific hashes(e.g., backup processes, production server processes, and the like).

FIG. 19B is a block diagram 1900B of a UI-based visualization ofanomalous log data, according to one embodiment. As shown in FIG. 19B,unusual activity 1950 (e.g., potentially anomalous activity) includesone or more (visual) dashes for each hash (e.g., as shown in the leftside of the user interface in FIG. 19A). Selected items 1955(1)-(N) canvisually identify each log, timeseries data associated with the hash ofsaid log, and the (potentially malicious or anomalous) processassociated with the given log (e.g., Log 1, bad.exe, and 2019-07-0500:10, and the like, as shown in FIG. 19B).

FIG. 20 is a block diagram 2000 of a process for preparing and sendingvisualization data to a UI-based web application, according to oneembodiment. The process begins at 2005 by receiving UHP visualizerconfiguration data from user, and at 2010, accesses log data. At 2015,the process identifies and retrieves unique hash and key/value pairs,and at 2020, stores the unique hash and key/value pairs in a database.At 2025, the process tracks new instances of each hash, and at 2030,adds to the timeseries data in the database. The process ends at 2035 bysending visualization data to a UI web application (e.g., UHP-timelinevisualization interface 1905).

Example of Applying Markovian Analysis to Threat Hunting

Malware tends to really like (and favor) randomized strings (e.g., aseries of number and letters that have no pattern—for instance, a16-character unique output). If a piece of malware is not randomizingthe information about itself (e.g., the malware) that appears in log,defenders (e.g., SOC analysts) can quickly learn to create rules thatcan match against an entire family (of malware). Static strings canbecome (not great) indicators of compromise (IOCs) and SOC analysts canalso use static strings to link families of malware together. To avoidthe foregoing, malware tends to (fully or partially) randomizeeverything about itself.

For real time detection, string IOCs are rather weak and instead, rulesare set on specific process behaviors that have been used in previousmalicious attacks. However, in threat hunting, SOC analysts typicallymanipulate and search through data to detect anomalies that may beindicative of attacker behavior (e.g., in case either the detectionshave somehow failed or the malware is utilizing a new behavioraltogether).

One method to detect malware during hunting involves gathering a largeset of log data and “stacking” on fields (e.g., counting the uniquevalues for a particular field). For a SOC analyst, a positive (or good)sign of malware when analyzing stack data involves finding a path thatincludes unusual or randomized strings (e.g.,C:\ProgramData\ewposnfpwe\cxnvxio.exe—which can be a dead giveaway).However, detecting these unusual or randomized strings in a sea oflegitimate paths can be extremely challenging to perform (e.g.,manually). Even worse, there are certain pieces of legitimate softwarethat can use some form of randomized strings (e.g.,C:\users\user61\appdata\local\apps\2.0\0cyyyw94.wt9\lcw31498.at7\dell..tion_831211ca63b981c5_0008.000b_165622fff4cd0fc1\dellsystemdetectexe).

Such legitimate paths often use some sort of consistent pattern (e.g., acombination of version numbers, hashes, and universally uniqueidentifiers (UUIDs)). One can manually suggest “normalizations” forlarge sets of paths. For example, normalizations can be manually createdfor the following paths (e.g., as shown in FIG. 21 ):

-   -   C:\users\user1232\appdata\local\microsoft\onedrive\onedrive.exe    -   C:\users\admin-dave\appdata\local\microsoft\onedrive\onedrive.exe    -   C:\users\user12\appdata\local\microsoft\onedrive\onedrive.exe    -   C:\users\user821\appdata\local\microsoft\onedrive\onedrive.exe    -   C:\users\software_svc\appdata\local\microsoft\onedrive\onedrive.exe

By creating regex (regular expression(s)) to detect and replace theusers with % USERNAME %, the number of unique lines can be condenseddown to: C:\users\% USERNAME%\appdata\local\microsoft\onedrive\onedrive.exe.

Therefore, with respect to malware and threat hunting (in the context ofrandom strings), there are at least two existing technology-relatedproblems that require technology-related solutions: (1) asystem/method/approach that can automatically identify paths withunusual or randomized strings and present them to a SOC analyst aboveother(s) (other strings) and (2) a system/method/approach that canautomatically suggest regex normalizations for large amounts ofrepetitive data.

Markovian analysis engine 175 configures, implements, and performsMarkovian analysis with respect to threat hunting in at least threestages. In the first stage, a variation of Markovian string analysisusing a combination of techniques (one or more than are unknown in theart) is performed. For example, a variety of Markov prediction modelsare used including variations of Ngram and skip-grams. However, in oneembodiment, these various Markov prediction models are applied at bothan individual character level (e.g., because of paths and command linearguments) and at a folder/file name level. In addition, a variety ofRegex character replacements are performed for pre-processing to developMarkov models for picking up common patterns of special characters(e.g., globally unique identifiers (GUIDs).

In some embodiments, the foregoing wide variety of Markov models can beapplied to either individual key/values in a log to generate theprobabilistic likelihood of particular values (e.g., file paths, and thelike). In the alternative, such analysis can be performed on variouskey/values in a log to retrieve a probabilistic likelihood of aparticular log.

In the second stage, and in other embodiments, a UI component forapplying a heatmap based on probability to characters in a string isconfigured and implemented. For example, characters that have a highlikelihood versus a low likelihood can be colored differently. Whenreviewing paths and logs with low probabilistic likelihoods, it may notalways be apparent to a SOC analyst why that (given) path is unlikely.Therefore, highlighting characters that are unlikely permits the SOCanalyst to readily identify what is irregular. In the third stage, andin some embodiments, Markovian analysis engine 175 implements a systemthat suggests normalization or Regex replacements for particular sets ofpaths (e.g., at least because GUIDs and other common patterns can createsignificant noise into lists of unique values).

In one embodiment, Markovian analysis engine 175 can find randomized orunusual executable names or paths in a hunt and the UI probabilityheatmap can permit a SOC analyst to readily identify character(s) in thepath that are unlikely or unusual. FIG. 21 is a block diagram 2100 of aMarkovian analysis engine, according to one embodiment. FIG. 21illustrates the aforementioned heatmap that visually identifiescharacters in a given path that are unusual or unlikely (e.g.,ewposnfpwe\cxnvxio, user 61 (and 0cyyyw94.wt9\lcw31498.at7,831211ca63b981c5, 0008.000b_165622fff4cd0fc1), 1232, admin-dave, 12,821, and software (shown bolded, underlined, or with a bigger font inFIG. 21 )). Further, the normalization suggestion provided by Markoviananalysis engine 175 can benefit other areas of threat hunting bynormalizing and therefore reducing the number of unique values.

A significant portion of Markovian analysis revolves around ngrams andskip grams. In one embodiment, Markovian analysis engine 175 performsMarkov(ian) analysis (e.g., statically modeling random processes, wherethe probabilistic distribution is obtained solely by observingtransitions from current state to next state) on paths. The followingapproaches are contemplated in certain embodiments with respect toapplying Markovian analysis to threat hunting (e.g., Markovian analysison paths):

In one embodiment, character/ngram chains (e.g., up to some maximumcharacter length N) are implemented: A series of adjacentcharacters—ngrams (e.g., for string “Windows”, starting at character“w”: n=1 is “wi”, n=2 is “win”, n=3 is “wind” etc. In anotherembodiment, a series of skip-grams are implemented (e.g., for string“Windows”, starting at character “w”: n=1 is “w_n”, n=2 is “w_d”, n=3 is“w_o”, etc.

In some embodiments, file/folder based chains (e.g., up to some maximumfolder depth I) are implemented: A series of adjacent folder/filesubstrings—ipaths (e.g., for“c:\windows\system32\windowspowershell\v1.0\powershell.exe” starting atsubstr “C:”: n=1 is “c:\windows”, n=2 is “c:\windows\system32”, n=3 is“c:\windows\system32\windowspowershell” etc. In other embodiments, aseries of skip paths is/are implemented (e.g., for“c:\windows\system32\windowspowershell\v1.0\powershell.exe” starting atsubstr “C:”: where, in one or more embodiments, n=1 is “c:\windows”, n=2is “c:\_\windows\system32”, n=3 is“c:\_\windows\system32\windowspowershell” etc.

In certain embodiments, negative ngram based chains are implemented(e.g., for “c:\windows\system32\windowspowershell\v1.0\powershell.exe”,starting at character “p”: n=−0 is “p_e”, n=−1 is “p_x”, n=−2 is “p_e”,n=−3 is “p_.” etc. In addition, variations of negative ngram basedchains for skip grams and for full paths are also contemplated.

In one embodiment, special consideration is provided for suggestingRegex, where a file/folder convention may have different characters thatcan be replaced by a regex suggestion. In this example, the characterspivoted on can only be not in the set [a-zA-z0-9] and there is anassumption that if there is a special naming convention forfiles/folders, the convention will not include differing numbers ofspecial characters (e.g., no folder cases such as “\foldername{b”,“\foldername{{b”, “\foldername{{{b”. In“C:\ProgramData{1B8BDEE8-91C9-542E-170F-CA6C8D4DD41A2}\nema.txt”:A-replace: replacing any [a-zA-Z0-0] length I with A[i], the path is“A:\AAAAAAAAAA{AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA}AAAA. AAA”,N-shrink: replacing any segment of [a-zA-Z0-0] with the count ofcharacters in set [a-zA-Z0-0], the path is “1:\11{8-4-4-4-12}\4.3” (itshould be noted that “11” and “12” are treated as single “characters”for ngrams), A-single: replacing any [a-zA-Z0-9] on n length with ‘A’,the path is “A:\A{A-A-A-A-A}A.A”.

In one embodiment, Markovian analysis engine 175 uses at least the threemethods described above and uses them to pre-process the training dataand then performs the ngram/skip gram/negative n gram chains on theprocessed paths. The foregoing approach can also be used for variousregex groups (e.g., [a-z], [A-Z], [0-9], [0-9A-F] (Hex), and the like.In this manner, processing can be parallelized.

FIG. 22 is a flowchart 2200 of a process for identifying anomalous logdata using Markovian prediction models, according to one embodiment. Theprocess begins at 2205 by performing Markovian string analysis byapplying multiple Markov prediction models to one or more key/values(KV-pairs) in a log. At 2210, the process applies a heatmap based on theprobability to characters in string using the a UI component. Theprocess ends at 2215 by identifying normalization or regex replacementsfor particular sets of paths.

Example Process to Perform Log Anomaly Detection, Visualization, andAnalysis

FIG. 23 is a flowchart 2300 of a process to perform log anomalydetection, according to one embodiment. The process begins at 2220 byperforming log anomaly detection using (one or more embodiments) of theLog 2Vec methodology described herein (e.g., provided by log anomalydetection engine 110 as shown in FIGS. 1A and 1B). At 2225, the processrefines anomalous data using Markovian analysis (e.g., performed byMarkovian analysis engine 175 as shown in FIG. 1B and FIG. 21 ). Theprocess ends at 2230 by displaying results using a UHP-timelinervisualization user interface (e.g., UHP-timeline visualization interface1905 as shown in FIG. 19A provided by UHP timeliner engine 170 as shownin FIG. 1B).

Example Computing Environment

FIG. 24 is a block diagram 2400 of a computing system, illustrating howa log anomaly detection, analysis, and visualization (DAV) engine 2465can be implemented in software, according to one embodiment. Computingsystem 2400 can include MDR server 105 and broadly represents any singleor multi-processor computing device or system capable of executingcomputer-readable instructions. Examples of computing system 2400include, without limitation, any one or more of a variety of devicesincluding workstations, personal computers, laptops, client-sideterminals, servers, distributed computing systems, handheld devices(e.g., personal digital assistants and mobile phones), networkappliances, storage controllers (e.g., array controllers, tape drivecontroller, or hard drive controller), and the like. In its most basicconfiguration, computing system 2400 may include at least one processor2455 and a memory 2460. By executing the software that executes loganomaly DAV engine 2465 (which includes log anomaly detection engine110, UHP timeliner engine 170, Markovian analysis engine 175, andsecurity operations engine 180 of FIG. 1B), computing system 800 becomesa special purpose computing device that is configured to perform loganomaly detection, analysis, and visualization (e.g., for threathunting, among other purposes/uses).

Processor 2455 generally represents any type or form of processing unitcapable of processing data or interpreting and executing instructions.In certain embodiments, processor 2455 may receive instructions from asoftware application or module that may cause processor 2455 to performthe functions of one or more of the embodiments described and/orillustrated herein. For example, processor 2455 may perform and/or be ameans for performing all or some of the operations described herein.Processor 2455 may also perform and/or be a means for performing anyother operations, methods, or processes described and/or illustratedherein. Memory 2460 generally represents any type or form of volatile ornon-volatile storage devices or mediums capable of storing data and/orother computer-readable instructions. Examples include, withoutlimitation, random access memory (RAM), read only memory (ROM), flashmemory, or any other suitable memory device. In certain embodimentscomputing system 2400 may include both a volatile memory unit and anon-volatile storage device. In one example, program instructionsimplementing log anomaly DAV engine 2465 (which includes log anomalydetection engine 110, UHP timeliner engine 170, Markovian analysisengine 175, and security operations engine 180) may be loaded intomemory 2460 (or memory 165 of FIG. 1B).

In certain embodiments, computing system 2400 may also include one ormore components or elements in addition to processor 2455 and/or memory2460. For example, as illustrated in FIG. 24 , computing system 2400 mayinclude a memory controller 2420, an Input/Output (I/O) controller 2435,and a communication interface 2445, each of which may be interconnectedvia a communication infrastructure 2405. Communication infrastructure2405 generally represents any type or form of infrastructure capable offacilitating communication between one or more components of a computingdevice.

Memory controller 2420 generally represents any type/form of devicecapable of handling memory or data or controlling communication betweenone or more components of computing system 2400. In certain embodimentsmemory controller 820 may control communication between processor 2455,memory 2460, and I/O controller 2435 via communication infrastructure2405. I/O controller 2435 generally represents any type or form ofmodule capable of coordinating and/or controlling the input and outputfunctions of a computing device. For example, in certain embodiments I/Ocontroller 2435 may control or facilitate transfer of data between oneor more elements of computing system 2400, such as processor 2455,memory 2460, communication interface 2445, display adapter 2415, inputinterface 2425, and storage interface 2440.

Communication interface 2445 broadly represents any type/form ofcommunication device/adapter capable of facilitating communicationbetween computing system 2400 and other devices and may facilitatecommunication between computing system 2400 and a private or publicnetwork. Examples of communication interface 2445 include, a wirednetwork interface (e.g., network interface card), a wireless networkinterface (e.g., a wireless network interface card), a modem, and anyother suitable interface. Communication interface 2445 may provide adirect connection to a remote server via a direct link to a network,such as the Internet, and may also indirectly provide such a connectionthrough, for example, a local area network. Communication interface 2445may also represent a host adapter configured to facilitate communicationbetween computing system 2400 and additional network/storage devices viaan external bus. Examples of host adapters include, Small ComputerSystem Interface (SCSI) host adapters, Universal Serial Bus (USB) hostadapters, Serial Advanced Technology Attachment (SATA), Serial AttachedSCSI (SAS), Fibre Channel interface adapters, Ethernet adapters, etc.

Computing system 2400 may also include at least one display device 2410coupled to communication infrastructure 2405 via a display adapter 2415that generally represents any type or form of device capable of visuallydisplaying information forwarded by display adapter 2415. Displayadapter 2415 generally represents any type or form of device configuredto forward graphics, text, and other data from communicationinfrastructure 2405 (or from a frame buffer, as known in the art) fordisplay on display device 2410. Computing system 2400 may also includeat least one input device 2430 coupled to communication infrastructure2405 via an input interface 2425. Input device 2430 generally representsany type or form of input device capable of providing input, eithercomputer or human generated, to computing system 2400. Examples of inputdevice 2430 include a keyboard, a pointing device, a speech recognitiondevice, or any other input device.

Computing system 2400 may also include storage device 2450 coupled tocommunication infrastructure 2405 via a storage interface 2440. Storagedevice 2450 generally represents any type or form of storage devices ormediums capable of storing data and/or other computer-readableinstructions. For example, storage device 2450 may include a magneticdisk drive (e.g., a so-called hard drive), a floppy disk drive, amagnetic tape drive, an optical disk drive, a flash drive, or the like.Storage interface 2440 generally represents any type or form ofinterface or device for transmitting data between storage device 2450,and other components of computing system 2400. Storage device 2450 maybe configured to read from and/or write to a removable storage unitconfigured to store computer software, data, or other computer-readableinformation. Examples of suitable removable storage units include afloppy disk, a magnetic tape, an optical disk, a flash memory device, orthe like. Storage device 2450 may also include other similar structuresor devices for allowing computer software, data, or othercomputer-readable instructions to be loaded into computing system 2400.For example, storage device 2450 may be configured to read and writesoftware, data, or other computer-readable information. Storage device2450 may also be a part of computing system 2400 or may be separatedevices accessed through other interface systems.

Many other devices or subsystems may be connected to computing system2400. Conversely, all of the components and devices illustrated in FIG.24 need not be present to practice the embodiments described and/orillustrated herein. The devices and subsystems referenced above may alsobe interconnected in different ways from that shown in FIG. 24 .Computing system 2400 may also employ any number of software, firmware,and/or hardware configurations. For example, one or more of theembodiments disclosed herein may be encoded as a computer program (alsoreferred to as computer software, software applications,computer-readable instructions, or computer control logic) on acomputer-readable storage medium. Examples of computer-readable storagemedia include magnetic-storage media (e.g., hard disk drives and floppydisks), optical-storage media (e.g., CD- or DVD-ROMs),electronic-storage media (e.g., solid-state drives and flash media), andthe like. Such computer programs can also be transferred to computingsystem 2400 for storage in memory via a network such as the Internet orupon a carrier medium.

The computer-readable medium containing the computer program may beloaded into computing system 2400. All or a portion of the computerprogram stored on the computer-readable medium may then be stored inmemory 2460, and/or various portions of storage device 2450. Whenexecuted by processor 2455, a computer program loaded into computingsystem 2400 may cause processor 2455 to perform and/or be a means forperforming the functions of one or more of the embodimentsdescribed/illustrated herein. Alternatively, one or more of theembodiments described and/or illustrated herein may be implemented infirmware and/or hardware, or via one or more machine learning model(s)(e.g., to perform log anomaly detection, visualization, and analysis forthreating hunting, among other uses).

Example Networking Environment

FIG. 25 is a block diagram of a networked system, illustrating howvarious computing devices can communicate via a network, according toone embodiment. Network 155 generally represents any type or form ofcomputer network or architecture capable of facilitating communicationbetween MDR server 105 and clients 145(1)-(N). For example, network 155can be a Wide Area Network (WAN) (e.g., the Internet), a Storage AreaNetwork (SAN), or a Local Area Network (LAN).

Log anomaly DAV engine 2465 may be part of MDR server 105, or may beseparate (e.g., part of log anomaly DAV system 2505). All or a portionof embodiments discussed and/or disclosed herein may be encoded as acomputer program and loaded onto, stored, and/or executed by log anomalyDAV engine 2465, and distributed over network 155.

In some examples, all or a portion of log anomaly DAV system 2505 and/orMDR server 105 may represent portions of a cloud-computing ornetwork-based environment. These cloud-based services (e.g., software asa service, platform as a service, infrastructure as a service, etc.) maybe accessible through a web browser or other remote interface. Theembodiments described and/or illustrated herein are not limited to theInternet or any particular network-based environment.

Various functions described herein may be provided through a remotedesktop environment or any other cloud-based computing environment. Inaddition, one or more of the components described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, log anomaly DAV engine 2465 maytransform the behavior of MDR server 105 or log anomaly DAV system 2505to perform log anomaly detection, visualization, and analysis for threathunting by SOC analysts in a managed detection and response context(e.g., in cybersecurity computing environments).

Although the present disclosure has been described in connection withseveral embodiments, the disclosure is not intended to be limited to thespecific forms set forth herein. On the contrary, it is intended tocover such alternatives, modifications, and equivalents as can bereasonably included within the scope of the disclosure.

What is claimed is:
 1. A computer-implemented method, comprising:performing, by one or more hardware processors with associated memorythat implement a context-based anomalous log data identification system:receiving log data comprising a plurality of logs; generating a contextassociated training dataset, comprising splitting a string in a log ofthe plurality of logs into a plurality of split strings, generating acontext association between each of the plurality of split strings and aunique key that corresponds to the log, and generating an input/output(I/O) string data batch comprising I/O string data for each split stringin the log by training each split string against every other splitstring of the plurality of split strings in the log; and training acontext-based anomalous log data identification model using the I/Ostring data batch comprising a list of unique strings in the contextassociated training dataset and according to a machine learningtechnique, wherein the training tunes the context-based anomalous logdata identification model to classify or cluster a vector associatedwith a new string in a new log that is not part of the plurality of logsas anomalous, training the context-based anomalous log dataidentification model to perform cluster analysis is based on whether anexecutable that is part of the process information is a good executablethat is part of a bad path, and the good executable and the bad path arepre-identified based at least on a classifier prior to performing thecluster analysis.
 2. The computer-implemented method of claim 1, furthercomprising: generating a dense vector for the log.
 3. Thecomputer-implemented method of claim 2, wherein generating the densevector for the log comprises: accessing the list of unique splitstrings, and averaging a plurality of vectors comprising at least onevector for each unique split string in the list of unique split strings,and the dense vector indicates a mapping of each unique split string inthe list of unique split strings to the dense vector being trained. 4.The computer-implemented method of claim 3, further comprising: trainingthe context-based anomalous log data identification model withadditional I/O string data generated by the context-based anomalous logdata identification system for each log of the plurality of logs.
 5. Thecomputer-implemented method of claim 1, wherein the log data comprisesprocess information associated with one or more computing systemsgenerating the log data, and the process information comprises aplurality of process names/hashes.
 6. The computer-implemented method ofclaim 5, wherein training the context-based anomalous log dataidentification model to perform cluster analysis is based at least on anumber of occurrences of a process name/hash of the plurality of processnames/hashes in the log.
 7. A non-transitory computer readable storagemedium comprising program instructions executable to: perform, by one ormore hardware processors with associated memory that implement acontext-based anomalous log data identification system: receive log datacomprising a plurality of logs; generate a context associated trainingdataset, comprising splitting a string in a log of the plurality of logsinto a plurality of split strings, generating a context associationbetween each of the plurality of split strings and a unique key thatcorresponds to the log, and generating an input/output (I/O) string databatch comprising I/O string data for each split string in the log bytraining each split string against every other split string of theplurality of split strings in the log; and train a context-basedanomalous log data identification model using the I/O string data batchcomprising a list of unique strings in the context associated trainingdataset and according to a machine learning technique, wherein thetraining tunes the context-based anomalous log data identification modelto classify or cluster a vector associated with a new string in a newlog that is not part of the plurality of logs as anomalous, training thecontext-based anomalous log data identification model to perform clusteranalysis is based on whether an executable that is part of the processinformation is a good executable that is part of a bad path, and thegood executable and the bad path are pre-identified based at least on aclassifier prior to performing the cluster analysis.
 8. Thenon-transitory computer readable storage medium of claim 7, furthercomprising: generating a dense vector for the log.
 9. The non-transitorycomputer readable storage medium of claim 8, wherein generating thedense vector for the log comprises: accessing the list of unique splitstrings, and averaging a plurality of vectors comprising at least onevector for each unique split string in the list of unique split strings,and the dense vector indicates a mapping of each unique split string inthe list of unique split strings to the dense vector being trained. 10.The non-transitory computer readable storage medium of claim 9, furthercomprising: training the context-based anomalous log data identificationmodel with additional I/O string data generated by the context-basedanomalous log data identification system for each log of the pluralityof logs.
 11. The non-transitory computer readable storage medium ofclaim 7, wherein the log data comprises process information associatedwith one or more computing systems generating the log data, and theprocess information comprises a plurality of process names/hashes. 12.The non-transitory computer readable storage medium of claim 11, whereintraining the context-based anomalous log data identification model toperform cluster analysis is further based at least on a number ofoccurrences of a process name/hash of the plurality of processnames/hashes in the log.
 13. A system comprising: one or moreprocessors; and a memory coupled to the one or more processors, whereinthe memory stores program instructions executable by the one or moreprocessors to: perform, by one or more hardware processors withassociated memory that implement a context-based anomalous log dataidentification system: receive log data comprising a plurality of logs;generate a context associated training dataset, comprising splitting astring in a log of the plurality of logs into a plurality of splitstrings, generating a context association between each of the pluralityof split strings and a unique key that corresponds to the log, andgenerating an input/output (I/O) string data batch comprising I/O stringdata for each split string in the log by training each split stringagainst every other split string of the plurality of split strings inthe log; and train a context-based anomalous log data identificationmodel using the I/O string data batch comprising a list of uniquestrings in the context associated training dataset and according to amachine learning technique, wherein the training tunes the context-basedanomalous log data identification model to classify or cluster a vectorassociated with a new string in a new log that is not part of theplurality of logs as anomalous, training the context-based anomalous logdata identification model to perform cluster analysis is based onwhether an executable that is part of the process information is a goodexecutable that is part of a bad path, and the good executable and thebad path are pre-identified based at least on a classifier prior toperforming the cluster analysis.
 14. The system of claim 13, furthercomprising: generating a dense vector for the log, wherein generatingthe dense vector for the log comprises: accessing the list of uniquesplit strings, and averaging a plurality of vectors comprising at leastone vector for each unique split string in the list of unique splitstrings, and the dense vector indicates a mapping of each unique splitstring in the list of unique split strings to the dense vector beingtrained.
 15. The system of claim 14, further comprising: training thecontext-based anomalous log data identification model with additionalI/O string data generated by the context-based anomalous log dataidentification system for each log of the plurality of logs.
 16. Thesystem of claim 13, wherein the log data comprises process informationassociated with one or more computing systems generating the log data,and the process information comprises a plurality of processnames/hashes.
 17. The system of claim 16, wherein training thecontext-based anomalous log data identification model to perform clusteranalysis is based at least on a number of occurrences of a processname/hash of the plurality of process names/hashes in the log.